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EXECUTIVE SUMMARY 



INTRODUCTION 

This report discusses the findings from an evaluation study of the Learning Without 
Limits program in the fifth grade at Riverdale Elementary in the Shelby County 
Schools district. The overall purpose of the evaluation was twofold: (a) to provide 
formative evaluation data to Riverdale School to use as a basis for improvement 
planning and to document their accomplishments to demonstrate progress; and (b) 
to provide cumulative evidence of the implementation progress and outcomes of 
the participant classes as well as identification of exemplary practice. 

The Learning Without Limits program was a pilot project designed to determine the 
impact of changing the ways students learn and teachers instruct in a technology- 
rich environment. The original Learning Without Limits concept included a laptop 
for each student in fifth grade with dedicated technology training for the fifth grade 
teachers. Unfortunately, costs and lack of significant grant funding prohibited the 
implementation of the project to this extent. 

Instead, the context for the Learning Without Limits program and this evaluation 
consisted of four fifth grade classes in which two Apple iBook wireless laptop carts 
were shared among the four fifth grade teachers' classrooms. In addition, the four 
teachers each received a personal Apple PowerBook laptop to use during the 
initiative and individually focused their professional development opportunities on 
technology-related training offered through Shelby County Schools. 

EVALUATION QUESTIONS 

The Learning Without Limits program evaluation was structured around five primary 
research questions that focused on classroom practices, degree and type of 
technology use, academically focused time, student engagement, teacher 
technology skills, teacher attitudes toward technology, as well as student and 
teacher reactions to the program. 

1 . In what ways has the effectiveness of instruction through the use of student 
laptop computer been impacted? 

2. To what degree and in what ways have teachers integrated technology with 
classroom instruction? 

3. To what degree do Learning Without Limits teachers use methodologies that 
stress higher-order learning and student-centered learning activities? 

4. To what degree has the Learning Without Limits program impacted teacher 
attitudes toward technology? 
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5. What factors appear most instrumental in determining schools' success at 
achieving the goals and overall implementation of the Learning Without Limits 
program? 

DESIGN 

The evaluation design was based on both quantitative and qualitative data 
collected from classroom observations, teacher surveys, and focus groups with 
teachers and students. The four fifth grade teachers and their intact classrooms at 
Riverdale Elementary participated in the evaluation. 

Instrumentation 

Five instruments and focus group interviews were used to collect the evaluation 
data (three classroom observation measures, two teacher surveys and four 
interviews). 

Classroom Observation Measures 

Observations were made focusing on targeted classes (scheduled visits) using three 
instruments. 

School Observation Measure (SOM): Examines frequency of usage of 24 
instructional strategies. 

Survey of Computer Use (SCU): Examines availability and student use of 
technology and software applications. 

Rubric for Student-Centered Activities (RSCA): Rates the degree of learner 
engagement in cooperative learning, project-based learning, higher-level 
questioning, experiential/hands-on learning, student independent inquiry/research, 
student discussion, and students as producers of knowledge using technology. 

Surveys 

Teacher Technology Questionnaire (TTQ): Collects teacher perceptions of 
computers and technology. 

Technology Skills Assessment (TSA): Assesses the perceived technological abilities 
of the teachers in these areas: Computer Basics, Software Basics, Multimedia 
Basics, Internet Basics, Advanced Skills, Using Technology for Learning, and 
Attitudes toward the Laptop Program. 

Focus Croups 

Focus groups were conducted with both teachers and students at the beginning of 
the Learning Without Limits initiative in Fall 2003 and again at the conclusion of 
the academic year in May 2004. From audio recordings and facilitator notes, 
themes were derived from the participants. 
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Procedure 

Data for this evaluation study were collected primarily in Spring 2004. The SOM, 
SCU and RSCA were completed for the targeted observations. These consisted of 
prearranged one-hour sessions in which the fifth grade teachers demonstrated a 
prepared lesson using technology. Observation notes forms were completed every 
' 1 5 minutes of the lesson. A total of 9 visits across the four fifth grade classrooms 
were completed. The teacher surveys (TTQ and TSA) were administered in May 
2004 prior to a focus group interview. Focus group interviews were conducted 
with students and teachers in Fall 2003 and again in May 2004. 

RESULTS 

Below is a brief summary of the results grouped by Classroom Observation Results 
and Survey Results. 

Classroom Observation Measures 

The data for 9 classroom observations were collected with SOMs, SCUs and RSCAs 
during prearranged sessions in which teachers implemented a lesson using 
technology. Results from each measure are described in the sections below. 

School Observation Measure (SOM®) 

The SOM revealed nine instructional strategies that were observed during the 
targeted observations. These strategies were observed during 1 1 .1 % of the visits to 
1 00% of the visits. Four strategies were observed in over 50% of the visits: 
project-based learning, technology as a learning tool, teacher acting as 
coach/facilitator and independent seatwork. Three more instructional strategies 
were observed in at least 30% of the visits: cooperative/collaborative learning, 
independent inquiry/research and higher-level instructional feedback. 

Academically focused class time and student engagement were observed to be high 
1 00% of the time. 

Survey of Computer Use (SCU®) 

Three computer activities were observed in at least 40% of the classroom visits: 
Internet browsers, draw/paint/graphics and electronic presentations. Computer 
activities were observed in all subject areas and meaningful and very meaningful 
computer applications were observed in at least 30% of the classrooms. 

Rubric for Student-Centered Activities (RSCA®) 

Five of the seven activities noted on the RSCA were observed during visits. Four of 
these activities were observed during at least 40% of the time: project-based 
learning, students as producers of knowledge, cooperative learning, and 
independent inquiry/research. Notably, project-based learning was observed 
during all observations and students as producers of knowledge were observed 
during almost 90% of the visits. Technology was used to support three of these 
strategies: cooperative learning, project-based learning and independent 
inquiry/research. 
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Survey Results 

Two surveys (TTQ and TSA) were administered to the teachers prior to a focus 
group interview in May 2003. Results of the three surveys are presented below. 

Teacher Technology Questionnaire (TTQ) 

The fifth grade teachers responded very positively to the Learning Without Limits 
program. This indicates the teachers feel this initiative has had a positive impact 
on classroom instruction, students, their readiness to integrate technology, the 
school and district's overall support for technology, appropriate technical support 
and a positive attitude toward the laptop program in general. 

Technology Skills Assessment (TSA) 

The TSA revealed very high levels of confidence by the fifth grade teachers to use 
technology throughout all areas. Of the 47 items on the TSA, the teachers rated 30 
of the items as able to be completed very easily. Only 4 questions' means were 
rated below 2 ("Somewhat able to complete the task"). 

Focus Group Results 

Two focus group interviews each were conducted with the four fifth grade teachers 
and approximately 8 to 1 0 of the fifth graders. Initial interviews were conducted in 
Fall 2003 and follow-up interviews in May 2004. 

Teacher Focus Group 

The teachers primarily discussed two themes: (a) computer use and (b) pressures 
and concerns. The teachers discussed their uses and their students' uses of the 
laptops and computer applications. Responsibility for care of the laptops was 
included as part of the instruction, as well as media literacy skills. Pressures and 
concerns regarding technical issues, confidence and administrative pressures 
appeared to have been reconciled by the spring semester. However, concerns over 
the scope of the curriculum and meeting content standards continued to challenge 
the teachers. 

Student Focus Group 

Students' discussions centered on two themes: (2) computer use and (b) students 
that benefit from using laptops. Students described the types of projects they had 
completed using the laptops. Using the laptops had made their work "easier," 

"fun" and required less "writing." But they recognized that some activities were 
"harder" on the laptops as well. Barriers to computer use included technical 
problems, such as system crashes, and logistical issues, such as lack of "battery 
power." The message from the teachers regarding responsibility using the laptops 
was also discussed. The students who benefited most from using the laptops were 
another primary topic. The students felt "fast learners," students who were " into 
computers" and "students who are not as smart" benefited the most. They also felt 
students who were indifferent and lackadaisical, called "goofers," benefited least. 
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CONCLUSIONS & RECOMMENDATIONS 

The conclusions of the study will be presented in association with each of the 
major research questions in the respective sections below. Recommendations are 
also included, where appropriate, to suggest improvements. 

In what ways has the effectiveness of instruction through the use of student laptop 
computer been impacted ? 

While it is difficult to determine increases without baseline, or beginning, data, 
SOM results indicate extensive uses of cooperative/collaborative learning, project- 
based learning and the teachers acting as coaches or facilitators. Results from the 
SCU indicate extensive uses of productivity tools, specifically draw/paint/graphics 
and electronic presentations, and Internet research with Internet browsers. SCU 
results also suggest wide use across the content areas. Finally, the overall 
meaningfulness of the computers was observed to be extensive in approximately 
one-third to fifty percent of the classroom visits. The results from the SOM coupled 
with these results from the SCU point to activities that result in meaningful uses of 
computers that were based on meaningful problems, required critical thinking skills 
and used computer applications to locate, process and/or manipulate information. 
Despite the limited scope of technology tools, those tools observed were seen to 
meaningfully integrate technology to enhance student learning. 

Moreover, data from the RSCA revealed that teachers used technology with over 
40% of their student-centered learning activities. This finding is not surprising 
given that the observations were conducted with targeted lessons, where the 
teachers were asked to demonstrate technology integration. However, specifically 
notable is that technology was observed in every instance with project-based 
learning. This is a significant accomplishment given the focus of the initiative on 
using technology to impact teaching and learning. Plus, technology for student use 
is often employed best during more unstructured learning such as projects. 

It is important to indicate that during two of the observations as noted in the SCU, 
the school network access was intermittent. As the Learning Without Limits 
program continues and potentially expands, school network reliability will affect 
the effectiveness of instruction with the increased dependence on digital resources, 
such as the Internet, email and school servers. 

To what degree and in what ways have teachers integrated technology with 
classroom instruction ? 

The original Learning Without Limits program included directed technology 
integration training for the fifth grade teachers; this was not implemented due to 
cost and lack of grants support. While no extensive Learning Without Limits 
training was conducted, the fifth grade teachers participated in a workshop on 
using Microsoft PowerPoint in Fall 2003 and informal discussions about classroom 
management with laptops during Spring 2003. Moreover, the teachers through 
focus group interviews indicated they had participated in professional development 
workshops offered through Shelby County Schools. They also relied on one 
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another to extend their expertise, creating a learning community. So, primarily the 
teachers used their educational philosophies and pedagogy to envision effective 
technology integration. 

As discussed above, SOM and SCU data indicate the teachers implemented 
instructional activities that resulted in meaningful uses of computers and required 
critical thinking skills. Productivity tools (e.g., word processing, 
draw/paint/graphics, electronic presentations and Timeliner®) and Internet research 
tools (e.g., Netscape Navigator) were used in all subject areas and educational 
software (e.g., drill and practice) was used in Mathematics. In addition, SOM and 
RSCA data revealed that the teachers employed project-based learning during all 
(1 00%) of the targeted classroom visits to varying degrees (limited to strong 
applications) and technology was used in every instance (100%). These results are 
suggestive that the Learning Without Limits program is effectively impacting 
technology integration efforts at Riverdale Elementary. 

To extend the teachers' effectiveness and expertise with technology integration and 
the instructional strategies that support technology, the teachers should participate 
in professional development targeted at pedagogy and technology integration, such 
as the training proposed in the original Learning Without Limits proposal plan. This 
training may offer the teachers a broader vision for technology integration with the 
school curricula and provide added support for changing teaching and learning 
with laptops. 

To what degree do Learning Without Limits teachers use methodologies that stress 
higher-order learning and student-centered learning activities? 

In almost 90% (88.9%) of the targeted classroom visits, teachers were observed 
extensively to be acting as a coach or facilitator of learning. Other activities 
indicative of critical thinking and student engagement were seen in over 30% of 
the visits. Cooperative/collaborative learning, which was observed extensively in 
33.3% of the visits, was observed to be a somewhat strong or strong application in 
over 40% of the observations (44.4%). Project-based learning, which was 
observed in 1 00% of the visits, was observed to be somewhat strong or strong 
application in almost 75% of the visits (77.7%). Finally, independent 
inquiry/research, which was observed in over 40% of the visits (44.4%), was 
observed to be a somewhat strong or strong application in approximately 1 0% of 
the visits (1 1 .1 %). These data indicate some use of non-traditional, or more 
student-centered, instructional methods. However, professional development 
efforts or goals focus more on these strategies to increase the frequency with which 
they are used. 

To what degree has the Learning Without Limits program impacted teacher 
attitudes toward technology? 

The fifth grade teachers are enthusiastic about the Learning Without Limits program. 
Succinctly, the teachers feel the program has positively impacted their classroom 
instruction and positively impacted the fifth grade students. Moreover, the teachers 
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feel they are ready to integrate technology into their instruction. This is 
corroborated by the focus group interviews as the teachers discussed their 
"confidence" with the laptops and reconciliation with previous intrinsic and 
extrinsic pressures. The TSA highlighted the teachers' expertise in computer basics, 
software basics, multimedia basics, Internet basics, advanced skills and using 
technology to support learning. Personal professional development objectives or 
grade level professional development goals should consider the few technology 
skills, such as Boolean strategy searches, electronic communications other than 
email and electronic teaching portfolios, that received mean scores below 2 
(between "Not at All" and "Somewhat"). These skills should be evaluated for their 
value to the teachers, students and relevance to the program as the initiative 
continues. 

Finally, the teachers felt they have the support of parents, the community, the 
administration and the technical support necessary to be effective with technology 
integration and improve student learning. 

What factors appear most instrumental in determining schools' success at 
achieving the goals and overall implementation of the Learning Without Limits 
program ? 

From the results of the observation instruments (i.e., SOM, SCU, RSCA), teacher 
surveys (i.e., TTQ, TSA) and focus group interviews (with both teachers and 
students), the fifth grade teachers and students are making marked progress to their 
goals. Below is a list of key factors from this evaluation that appear to have 
influenced the progress being made toward achieving the Learning Without Limits 
program implementation and goals. These factors represent high frequencies of 
occurrence and quality of implementation in observations. As seen, the factors 
cross key elements that are critical for program success: 

■ Types of instructional practices 

■ Quality of instructional practices 

■ Teacher attitudes and beliefs regarding technology integration 

■ Teacher technology skills 

Overall, the triangulation of positive results suggest a strong synergy among the 
diverse factors that appear to have contributed to the Learning Without Limits goals 
during the 2003-2004 school year. The fifth grade teachers and administration 
should be proud of the progress the program has made in a short time. As seen, the 
school had positive teacher technology competence and confidence, used 
instructional strategies that centered on and facilitated student learning and 
employed classroom practices that engaged students in meaningful technology- 
supported activities. 
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"LEARNING WITHOUT LIMITS" 
EVALUATION REPORT 

This report discusses the findings from an evaluation study of the Learning Without 
Limits program in the fifth grade at Riverdale Elementary in the Shelby County 
Schools district. The overall purpose of the evaluation was twofold: (a) to provide 
formative evaluation data to Riverdale School to use as a basis for improvement 
planning and to document their accomplishments to demonstrate progress; and (b) 
to provide cumulative evidence of the implementation progress and outcomes of 
the participant classes as well as identification of exemplary practice. 

The Learning Without Limits program was a pilot project designed to determine the 
impact of changing the ways students learn and teachers instruct in a technology- 
rich environment. The original Learning Without Limits concept included a laptop 
for each student in fifth grade with dedicated technology training for the fifth grade 
teachers. Unfortunately, costs and lack of significant grant funding prohibited the 
implementation of the project to this extent. 

Instead, the context for the Learning Without Limits program and this evaluation 
consists of four fifth grade classes in which two Apple iBook wireless laptop carts 
were shared among the four fifth grade teachers' classrooms. In addition, the four 
teachers each received a personal Apple PowerBook laptop to use during the 
initiative and individually focused their professional development opportunities on 
technology-related training offered through Shelby County Schools. 

This evaluation concentrated on the progress of the four fifth grade classes in 
achieving the Learning Without Limits goals and objectives. As a means of 
examining this progress, data were collected primarily at the end-of-year. 

Suggestions for improvement are offered on the basis of the findings. Specific 
evaluation questions that guided the methodology, data collection and reporting 
are listed in the next section. 

EVALUATION QUESTIONS 

1 . In what ways has the effectiveness of instruction through the use of student 
laptop computer been impacted? 

2. To what degree and in what ways have teachers integrated technology with 
classroom instruction? 

3. To what degree do Learning Without Limits teachers use methodologies that 
stress higher-order learning and student-centered learning activities? 

4. To what degree has the Learning Without Limits program impacted teacher 
attitudes toward technology? 
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5. What factors appear most instrumental in determining schools' success at 
achieving the goals and overall implementation of the Learning Without 
Limits program? 

EVALUATION DESIGN & MEASURES 

To address the above questions, the basic evaluation design employed targeted 
visits to each fifth grade classroom by trained researchers. Nine classroom visits 
were conducted during Spring 2004. This yielded nine end-of-year observation 
periods. Reliability data indicate that consistent results are obtained with 6 visits. 

The researchers spent the major part of their time observing classrooms during 
Spring 2004, but also facilitated administration of teacher surveys in May 2004, as 
well as teacher and student focus groups in both Fall 2003 and Spring 2004. 
Descriptions of each measure and its administration procedure are given below. 

Instrumentation 

Five instruments were used to collect the evaluation data: three classroom 
observation measures and two teacher surveys. The instruments used in the study 
are fully developed and validated as appropriate. They are as follows: 

Classroom Observation Measures 

Trained observers conducted targeted classroom visits to collect frequency data 
regarding observed instructional practices. The visits were considered targeted 
because observations were scheduled in advance with randomly selected teachers 
(from those who participated in the program's technology training) who were 
instructed to deliver a lesson that integrates the use of technology. The data 
collection instruments were the School Observation Measure (SOM), the Survey of 
Computer Use (SCU), and the Rubric for Student-Centered Activities (RSCA). 

The SOM was used to collect data regarding overall classroom activities, the SCU 
for student use of computers, and the RSCA to capture more detailed information 
about student-centered activities during the targeted observations. The classroom 
observation instruments are described below. 

SOM. The SOM was developed to determine the extent to which different 
common and alternative teaching practices are used throughout an entire school 
(Ross, Smith, & Alberg, 1999). The standard, or whole-school SOM 0 procedure 
involves observers' visiting 10-12 randomly selected classrooms, for 15 minutes 
each, during a three-hour visitation period. The observer examines classroom 
events and activities descriptively, not judgmentally. Notes are taken relative to the 
use or nonuse of 24 target strategies. At the conclusion of the three-hour visit, the 
observer summarizes the frequency with which each of the strategies was observed 
across all classes in general on a data summary form. The frequency is recorded 
via a 5-point rubric that ranges from (0) Not Observed to (4) Extensively. Two 
global items use three-point scales (low, moderate, high) to rate, respectively, the 
use of academically focused instructional time and degree of student attention and 
interest. 
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Targeted observations were conducted in this evaluation to examine classroom 
instruction during prearranged one-hour sessions in which the teachers 
demonstrated a prepared lesson using technology. The notes forms were 
completed every 15 minutes of the lesson then were summarized on a SOM Data 
Summary Form. To triangulate the reliability of these results, multiple researchers 
observed class sessions. 

To ensure the reliability of data, observers receive a manual providing definitions of 
terms, examples and explanations of the target strategies, and a description of 
procedures for completing the instrument. The target strategies include traditional 
practices (e.g., direct instruction and independent seatwork) and alternative, 
predominately student-centered methods associated with educational reforms (e.g., 
cooperative learning, project-based learning, inquiry, discussion, using technology 
as a learning tool). The strategies were identified through surveys and discussions 
involving policy makers, researchers, administrators, and teachers, as those most 
useful in providing indicators of schools' instructional philosophies and 
implementations of commonly used reform designs (Ross, Smith, Alberg, & 

Lowther, 2001 ). 

To ensure the reliability of data, observers receive training, a manual providing 
definitions of terms, examples and explanations of the strategies, and a description 
of procedures for completing the instrument. After receiving the manual and 
instruction in a group session, each observer participates in sufficient practice 
exercises to ensure that his/her data are comparable with those of experienced 
observers. In a reliability study (Lewis, Ross, & Alberg, 1999), pairs of trained 
observers selected the identical overall response on the five-category rubric on 
67% of the items and were within one category on 95% of the items. Further 
results establishing the reliability and validity of 50A4® are provided in the Lewis et 
al. (1999) report. In a reliability study using Generalizability Theory, Sterbinsky 
(2003) found reliability at the .74 level for 5 SOMs conducted at a school. 

Reliability increased to .82 with 8 SOMs and to .85 with 10 SOMs conducted at a 
school. 

SCU. A companion instrument to SOM is the Survey of Computer Use (SCU) 
(Lowther & Ross, 2001 ). The SCU was completed as part of the SOM observation 
sessions, where SCU data was also recorded in 15-minute intervals and then 
summarized on an overall data form. 

The SCU was designed to capture exclusively student access to, ability with, and 
use of computers rather than teacher use of technology. Therefore, four primary 
types of data are recorded: (a) computer capacity and currency, (b) configuration, 
(c) student computer ability and (b) student activities while using computers. 
Computer capacity and currency is defined as the age and type of computers 
available for student use and whether or not Internet access is available. 
Configuration refers to the number of students working at each computer (e.g., 
alone, in pairs, in small groups). Student computer ability is assessed by recording 
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the number of students who are computer literate (e.g., easily used software 
features/menus, saved or printed documents) and the number of students who 
easily use the keyboard to enter text or numerical information. 

The next section of the SCU focuses on student use of computers with regard to: 
the types of activities, subject areas of activities, and software being used. The 
computer activities are divided into three categories based on the type of software 
tool: production tools, Internet/research tools, and educational software. Within 
each category, primary types of software are identified. For example, under 
Production Tools, the software includes: word processing, databases, spreadsheets, 
draw/paint/graphics, presentation (e.g., Microsoft PowerPoint™), authoring (e.g., 
KidPix™), concept mapping (e.g., Inspiration), and planning (Microsoft Project™). 
For the Internet/research tools, three types of software are included: Internet 
browser, CD reference materials, and communications (e.g., email, listservs, chat 
rooms). The Educational Software also has three types of software: 
drill/practice/tutorial, problem-solving (e.g., Riverdeep™) and process tools (e.g., 
Author's Toolkit™). With this type of recording system, several activities can be 
noted during the observation of one student working on a computer. For example, 
if a student gathered data from the Internet, created a graph from the data, then 
imported the graph into a PowerPoint presentation, the observer would record 
three types of software tools as being observed: Internet browser, spreadsheet, and 
presentation. This section ends by identifying the subject area of each computer 
activity. The categories include: language arts, mathematics, science, social 
studies, other, and none. The computer activities and software being used are 
summarized and recorded using a five-point rubric that ranges from (0) Not 
Observed to (4) Extensively observed. 

The final section of the SCU is an "Overall Rubric" designed to assess the degree to 
which the activity reflects "meaningful use" of computers as a tool to enhance 
learning. The rubric has four levels: 1 - Low-level use of computers, 2 - 
Somewhat meaningful, 3 - Meaningful, and 4 - Very meaningful. 

RSCA. The Rubric for Student-Centered Activities was developed by CREP 
(Lowther, Ross, & Plants, 2000) as an extension to SOM and SCU. The RSCA was 
used by observers to more closely evaluate the degree of learner engagement in 
seven selected areas considered fundamental to the goals of increasing student- 
centered learning activities (cooperative learning, project-based learning, higher- 
level questioning, experiential/hands-on learning, student independent 
inquiry/research, student discussion, and students as producers of knowledge using 
technology). These strategies reflect emphasis on higher-order learning and 
attainment of deep understanding of content and whether or not technology was 
utilized as a component of the strategy. Such learning outcomes seem consistent 
with those likely to be engendered by well-designed, real-world linked exercises, 
projects, or problems utilizing technology as a learning tool. Each item includes a 
two-part rating scale. The first is a four-point scale, with 1 indicating a very low 
level of application, and 5 representing a high level of application. The second is a 
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Yes/No option to the question: "Was technology used?" with space provided to 
write a brief description of the technology use. The RSCA was completed as part of 
SOM/SCU observation periods. 

Teacher Surveys 

Two surveys were used to collect impressions of the Learning Without Limits 
program: the Teacher Technology Questionnaire (TTQ), and the Technology Skills 
Assessment (TSA). Each of the four fifth grade teachers completes the surveys in 
May 2004. The surveys are described below. 



TTQ. The Teacher Technology Questionnaire is an instrument used to collect 
teacher perceptions of computers and technology. In the first section, teachers rate 
their level of agreement with 20 statements regarding five technology-related areas: 
impact on classroom instruction, impact on students, teacher readiness to integrate 
technology, overall support for technology in the school, and technical support. 
Items are rated with a five-point Likert-type scale that ranges from (1) Strongly 
Disagree to (5) Strongly Agree. A sixth section was added specifically to address 
perceptions of the laptop program also rated on a five-point scale from (1) Strongly 
Disagree to (5) Strongly Agree. 

TSA. The Technology Skills Assessment (TSA) is a 47-item survey that includes 
three-point Likert-type questions. The three-point questions are designed to assess 
the perceived technological abilities of the participants. All of the questions are 
arranged into six categories, which are aligned to the International Society for 
Technology in Education's (ISTE) National Educational Technology Standards 
(NETS). The categories of the survey are as follows: Computer Basics, Software 
Basics, Multimedia Basics, Internet Basics, Advanced Skills, and Using Technology 
for Learning. 

Focus Groups 

Student Reactions 

To increase understanding of the implementation processes and outcomes of the 
Learning Without Limits program, focus groups were conducted with eight to ten 
fifth grade students participating in the program. A semi-structured interview 
protocol was used in order to variation in the order and phrasing of the questions, 
as well as probes to specific individuals (Patton, 1990). These occurred in Fall 2003 
and in the Spring 2004. 

Teacher Reactions 

Like the students, teacher focus groups were conducted with all four fifth grade 
teachers twice, both in Fall 2003 and in Spring 2004. The teacher interview also 
used a semi-structured interview (Patton, 1990) and centered on three areas of 
questions: Using the Technology, Expectations and Reservations. 
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Data Collection 

Table 1 provides the type of measures, instruments, number collected, 
administration timeline and a brief description for each of the instruments 
described in the previous section. 

Table 1 . Data Collection Summary 



Type of 




Number 






Measure 


Instrument 


Collected 


Timeline 


Description 


Classroom 


SOM 


9 


Spring 2004 


Pre-arranged one-hour sessions 


Observations 


SCU 


9 




in which teachers demonstrated 




RSCA 


9 




a prepared lesson using 
technology 










Notes forms were completed 
every 1 5 minutes of the lesson. 
The four, 1 5-minute notes forms 
were aggregated onto a 
summary sheet. 


Surveys 


TTQ 


4 


Spring 2004 


Administered prior to focus 




TSA 


4 




group in May 2004 


Focus Group 


Semi-structured 


4 


Fall 2003 


Two student focus group 


Interviews 


interviews 




Spring 2004 


interviews were conducted with 
8-10 fifth grade students in fall 
and spring semesters. 










Two teacher focus group 
interviews were conducted with 
all 4 fifth grade teachers in fall 
and spring semesters. 
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RESULTS 



CLASSROOM OBSERVATION RESULTS 

The data for 9 classroom observations were collected with SOMs, SCUs, and 
RSCAs during prearranged sessions in which teachers implemented a lesson using 
technology. Results from each measure are described in the section below. This is 
followed with survey results. 

School Observation Measure (SOM®) 

Observed v. not observed. As indicated in the description of SOM, the observation 
procedure primarily focused on 24 instructional strategies using a five-point rubric 
(0 = not observed, 1 = rarely, 2 = occasionally, 3 = frequently, and 4 = 
extensively). In an initial analysis (Table 2), we computed the percentage of times a 
strategy was not observed (rubric category = 0, Not Observed) vs. observed 
(categories = 1-4 combined, Rarely, Occasionally, Frequently, Extensively). This 
offers schools the opportunity to note which instructional strategies were observed 
to determine if their goals are being met by what the researchers observed during 
lessons. 

Table 2. School Observation Measure (SOM) Proportion of Not Observed v. Observed Strategies 



Strategies 


Percent Not Observed 


Percent Observed 


Project-based learning 


0.0 


100.0 


Technology as a learning tool or resource (e.g. Internet 
research, spreadsheet or database creation, multi-media, CD 
ROM, Laser disk) 


0.0 


100.0 


Teacher acting as coach/facilitator 


11.1 


88.9 


Independent seatwork (self-paced worksheets, individual 
assignments) 


44.4 


55.6 


Cooperative/collaborative learning 


55.6 


44.4 


Independent inquiry/research on the part of the students 


55.6 


44.4 


Higher level instructional feedback (written or verbal) to 
enhance student learning 


66.7 


33.3 


Use of higher-level questioning strategies 


77.8 


22.2 


Direct instruction 


88.9 


11.1 



The most frequently observed strategies were "project-based learning" and 
"technology as a learning tool or resource." These were observed in every instance 
(1 00%). The next most often observed strategy was "teacher acting as 
coach/facilitator," observed in 88.9% of the classrooms. 
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Ratings. Table 3 presents the full, five-category breakdown of the SOM results. 
These results allow school to discern the extent to which instructional strategies 
were observed (i.e., from "Percent None" through "Percent Extensively"). Along 
with the results from Riverdale Elementary, three other measures are included: 
Laptop schools norms (N=10), Elementary schools national norms (N=624) and 
Secondary schools national norms (N=1,230). These scores are provided as a 
courtesy for comparison with Riverdale Elementary's results; however, no 
inferential statistics were computed. The results from the Laptop schools norms 
included targeted observations only, like those used in this evaluation; however, 
the Elementary and Secondary schools norms included both targeted and whole- 
school observations. 

Six strategies were discerned frequently or extensively in at least 30% of the 
observations. Four of these were observed in over 50% of the classroom visits: 
"project-based learning" (100%), "teacher acting as coach/facilitator" (88.9%), 
"independent seatwork" (55.6%) and "technology as a learning tool or resource 
(1 00%). The remaining two strategies were: "cooperative/collaborative learning" 
(33.3%) and "independent inquiry/research on the part of the student" (44.4%). 

In the final section of the SOM, "academically focused class time" and "level of 
student attention/interest/engagement" were both rated as high in every classroom 
visit (100%). 
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Table 3. School Observation Measure (SOM) Data Summary 

River = Riverdale Elementary, N = 9; Lap = Laptop School Norms, N=10; 

Elm = Elementary Schools National Norms, N=624; Sec = Secondary Schools National Norms, N=1,230 
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Independent seatwork (self-paced worksheets, 44.4 50.0 2.5 3.1 0.0 0.0 12.1 14.1 0.0 0.0 29.4 32.6 0.0 0.0 41.4 35.4 55.6 50.C 

individual assignments) 

Experiential, hands-on learning 100.0 100.0 23.2 30.2 0.0 0.0 40.0 43.0 0.0 0.0 26.1 20.4 0.0 0.0 8.7 5.0 0.0 0.0 

Systematic individual instruction (differential 100.0 100.0 67.4 77.3 0.0 0.0 22.7 13.5 0.0 0.0 6.9 5.7 0.0 0.0 2.9 2.9 0.0 0.0 

assignments geared to individual needs) 

















The extent to which each of the 

following was used or present in the Percent 

classroom. Percent None Percent Rarely Occasionally Percent Frequently Percent Extensively 
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Survey of Computer Use (SCU) 

As with the SOM, data from the SCU were collected during prescheduled lessons 
in which teachers were asked to use technology. A summary of the observation 
results is provided in Table 5 at the end of this section. Along with the results from 
Riverdale Elementary, two other measures are included: Laptop schools norms 
(N=10) and Elementary and Secondary schools national norms (N=1,867). These 
scores are provided as a courtesy for comparison with Riverdale Elementary's 
results; however, no inferential statistics were computed. The results from the 
Laptop schools norms included targeted observations only, like those used in this 
evaluation; however, the Elementary and Secondary schools norms include both 
targeted and whole-school observations. 

Computer Configuration. In all of the visits (1 00%), 1 1 or more computers were 
available for student use in the classrooms. All of the computers (1 00%) were 
observed to be up-to-date, and all the computers (100%) were connected to the 
Internet. It is important to note that during two of the pre-arranged visits, the 
school's internal network was intermittent. So while the computers were capable 
of accessing the intranet and Internet, students were unable to do so consistently. 

Student computer activities. An initial analysis compared those computer activities 
that were not observed (rating = 0) v. those observed (rating = 1 thru 4). These are 
listed in Table 4. Six activities were observed from 1 1 .1 % to 66.7% of the 
classroom visits. 

Table 4. Survey of Computer Use Proportion of Not Observed v. Observed Activities 



Student Computer Activities 


Percent Not Observed 


Percent Observed 


Internet Browser (e.g., Netscape) 


33.3 


66.7 


Draw/Paint/Graphics 


55.6 


44.4 


Presentation (e.g., MS PowerPoint) 


55.6 


44.4 


Word Processing 


77.8 


22.2 


Other (i.e., Timeliner) 


88.9 


11.1 


Drill/Practice/Tutorial 


88.9 


11.1 



Production tools used by students. As shown in Table 5, four of the nine different 
computer activities were observed: word processing, draw/paint/graphics, 
presentation and other (i.e., Timeliner®). Draw/paint/graphics and presentation 
activities were extensively observed in over 30% of the visits. With regard to the 
subject area focus, the majority of production tool activities were in language arts 
(33.3%) and mathematics (33.3%). Fewer production tool activities involved social 
studies (22.2%), and science (11.1%). 
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Internet/ research tools used by students. Internet browsers (e.g., Netscape 
Navigator) were the only research tools observed. Internet browsers were 
frequently and extensively observed in 44.4% of the classroom visits. The research 
activities covered all subject areas, however, the majority were focused on 
language arts (22.2%) and social studies (22.2%). The remainder of the subject 
areas was seen in only approximately 1 1 .1 % of the visits. 

Educational software use by students. Drill/practice/tutorial activities were the only 
educational software occasionally observed (1 1 .1 %). Mathematics software was 
the only subject observed in approximately 1 1 % (1 1.1%) of the visits. 

Testing software use by students. As seen, testing software was not observed during 
the classroom visits. 

Overall Meaningful Use of Computers. The culminating assessment on the SCU 
was the observer's evaluation of the meaningfulness of the way in which 
technology was integrated with teaching and learning. To do this, they were asked 
to indicate how often they observed computer activities at each level of the rubric; 
e.g., how often was very meaningful use of computers observed. As can be seen in 
Table 5, low levels of computer use and somewhat meaningful uses were rarely 
observed during the classroom visits (1 1 .1% each). However, meaningful uses of 
computers were occasionally observed in 11.1% of the visits while meaningful uses 
were extensively observed in over 50% of the visits (55.6%). Similarly, very 
meaningful uses of computers were extensively observed in over a third of the visits 
(33.3%). Combined, meaningful and very meaningful uses of computers were 
observed between approximately 30% (33.3%) and 65% (66.7%) of the visits. 
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Table 5. Survey of Computer Use (SCU) Data Summary 

Riverdale Elementary, N = 9; Laptop School Norms, N=40; National School Norms, N=1,867 



Computer Configuration (in percentages) 


Riverdale 


Laptop 


National 


Classrooms most frequently had the following number of computers 








or digital tools: 








0 


0.0 


0.0 


1.4 


1 


0.0 


0.0 


26.6 


2-4 


0.0 


0.0 


47.3 


5-10 


0.0 


0.0 


15.0 


1 1 or more 


100.0 


100.0 


9.7 


Classroom computers were most frequently: 








Up-to-date 


100.0 


100.0 


59.3 


Aging but adequate 


0.0 


0.0 


39.7 


Outdated/limited capacity 


0.0 


0.0 


0.5 


No computers were observed 


0.0 


0.0 


0.5 


Classroom computers were most frequently: 








Connected to the Internet 


100.0 


100.0 


89.4 


Not connected to the Internet 


0.0 


0.0 


10.1 


No computers were observed 


0.0 


0.0 


0.5 


Computer Use (in percentages) 


Riverdale 


Laptop 


National 


Classroom computers or digital tools were most frequently used by: 






Few (less than 1 0%) students 


0.0 


0.0 


33.5 


Some (about 1 0-50%) students 


0.0 


0.0 


20.2 


Most (about 51-90%) students 


0.0 


0.0 


9.2 


Nearly all (91-100%) students 


100.0 


100.0 


24.6 


Students did not use computers 


0.0 


0.0 


12.3 


Students most frequently worked with computers or digital tools: 








Alone 


88.9 


100.0 


66.5 


In pairs 


11.1 


20.0 


11.8 


In small groups 


0.0 


80.0 


5.9 


Students did not use computers 


0.0 


0.0 


15.8 


Student computer literacy skills were most frequently: 








Poor 


0.0 


0.0 


10.8 


Moderate 


0.0 


20.0 


34.5 


Very good 


100 


80.0 


31.5 


Not observed 


0.0 


0.0 


23.2 


Student keyboarding skills were most frequently: 








Poor 


0.0 


0.0 


13.4 


Moderate 


0.0 


30.0 


21.4 


Very good 


100 


70.0 


29.4 


Not observed 


0.0 


0.0 


35.8 
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Table 5. Survey of Computer Use (SCU) Data Summary continued 

River=Riverdale Elementary, N = 9; Lap=Laptop School Norms, N=40; Nat=National School Norms, N=1,867 
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Rubric for Student-Centered Activities (RSCA) 

Results address the percentage of sessions in which each RSCA strategy was 
observed at least once, the quality/depth of observed strategy applications, and the 
percentage of sessions in which technology was used with the observed strategy. 
Because the RSCA was used in targeted observations of lessons that were to include 
the use of technology to support learning, computer use was expected to be viewed 
at sometime during the lesson. 

Observed vs. not observed. The first question on each RSCA item asked whether 
or not the particular strategy was observed. To help schools determine if their 
goals and objectives are being met, it is easiest to note those activities that were 
observed at least once during the classroom visits. These activities observed are 
summarized in Table 6. As indicated in the SOM, project-based learning was 
observed during all classroom visits (100%). The next most often observed activity 
was "students as producers of knowledge" in 88.9% of the visits. 

Table 6. Rubric for Student-Centered Activities Proportion of Not Observed v. Observed Activities 



Student-Centered Activities 


Percent Not Observed 


Percent Observed 


Project-Based Learning 


0.0 


100.0 


Students as Producers of Knowledge 


11.1 


88.9 


Cooperative Learning 


55.6 


44.4 


Independent Inquiry / Research 


55.6 


44.4 


Higher-Level Questioning Strategies 


88.9 


11.1 



Ratings. Table 7 summarizes the complete results for the RSCA. These results allow 
school to discern the extent to which instructional strategies were observed (i.e., 
from "Not Observed" through "Strong Application"). All the activities specific to 
the RSCA except for student discussions and experiential hands-on learning were 
observed during the fifth grade classroom visits. Along with the results from 
Riverdale Elementary, two other measures are included: Laptop schools norms 
(N=10) and Elementary and Secondary schools national norms (N=1,788). These 
scores are provided as a courtesy for comparison with Riverdale Elementary's 
results; however, no inferential statistics were computed. The results from the 
Laptop schools norms included targeted observations only, like those used in this 
evaluation; however, the Elementary and Secondary schools norms include both 
targeted and whole-school observations. 

The most meaningful applications, that is those activities where somewhat strong 
and strong applications were observed in at least 30% of the classroom visits, 
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included cooperative learning (44.4%), project-based learning (77.7%) and 
students as producers of knowledge (77.8%). 

Technology use. Students used technology to support learning during cooperative 
learning (44.4%), project-based learning (100%) and independent inquiry/research 
(44.4%). It is important to note that technology was used in combination with 
cooperative learning and independent inquiry/research almost in almost half of the 
observations. Moreover, technology was used to support learning with project- 
based learning during 1 00% of the observations. All the results for technology use 
are summarized in Table 8. 
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Table 7. Rubric for Student Centered Activities (RSCA) Data Summary 

River=Riverdale Elementary, N = 9; Lap=Laptop School Norms, N=10; Nat=National School Norms, N=1,788 
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Table 8. Rubric for Student Centered Activities (RSCA) Data Summary 

Riverdale=Riverdale Elementary, N = 9; Laptop=Laptop School Norms, N=10; National=National School Norms, N=1,788 
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TEACHER SURVEY RESULTS 

Two surveys (TTQ, and TSA) were administered to the fifth grade teachers during 
prior to a focus group interview in May 2003. 

Teacher Technology Questionnaire (TTQ) 

The Teacher Technology Questionnaire (TTQ) was designed to capture teacher 
perceptions regarding their personal beliefs and practices regarding five areas: 
impact of technology on classroom instruction, impact of technology on students, 
teacher readiness to integrate technology, overall support for technology, technical 
support and attitude toward the laptop program. The four fifth grade teachers 
completed the survey. 

As seen in Table 9, mean scores for all six sections were between 4 ("Agree") and 5 
("Strongly Agree"). This indicates the teachers feel the Learning Without Limits 
program has had a positive impact on classroom instruction, a positive impact of 
technology on students, the teachers' readiness to integrate technology, the school 
and district's overall support for technology, appropriate technical support and a 
positive attitude toward the laptop program. 

Notable are the questions that the teachers responded in unison. Within the 
section on overall support for technology in the school, Questions 4 and 1 3 
address the parent, community and administrative support necessary for technology 
to impact teaching and learning, both with scores of 5 = "Strongly Agree." Within 
the technical support section, the teachers concurred on Question 2 that they could 
readily answer technology related questions with a score of 4 = "Agree." Within 
the section on attitudes toward the laptop program, Questions 21 and 23 represent 
the teachers' enthusiasm for the program and confidence about their abilities, both 
with scores of 5 = Strongly Agree. Also notable is Question 1, which received the 
lowest mean score of 3.75 between "Neither Disagree nor Agree" (3) and "Agree" 
(4). This question related to the maintenance of the school computers. This 
question while receiving the lowest mean score also had the largest amount of 
variance among the respondents (SD = 1 .26). 
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Table 9. Teacher Technology Questionnaire (TTQ) (N=4) 



Category and Related TTQ Items 


Means 


Standard 

Deviations 


Impact on Classroom Instruction 


14. 


My teaching is more student-centered when technology is integrated into the lessons. 


4.75 


.50 


16. 


I routinely integrate the use of technology into my instruction. 


4.25 


.50 


18. 


Technology integration efforts have changed classroom learning activities in a very 
positive way. 


4.50 


.58 


20. 


My teaching is more interactive when technology is integrated into the lessons. 


4.25 


.50 




Overall 


4.44 


.24 


Impact on Students 


3. 


The use of computers has increased the level of student interaction and/or collaboration. 


4.50 


.58 


8. 


The integration of technology has positively impacted student learning and 
achievement. 


4.75 


.50 


10. 


Most of my students can capably use computers at an age-appropriate level. 


4.50 


.58 


19. 


The use of technology has improved the quality of student work. 


4.75 


.50 




Overall 


4.63 


.32 


Teacher Readiness to Integrate Technology 


5. 


I know how to meaningfully integrate technology into lessons. 


4.50 


.58 


9. 


I am able to align technology use with my district's standards-based curriculum. 


4.00 


.82 


11. 


I have received adequate training to incorporate technology into my instruction. 


4.25 


.50 


12. 


My computer skills are adequate to conduct classes that have students using technology. 


4.50 


.58 




Overall 


4.31 


.38 


Overall Support for Technology in the School 


4. 


Parents and community members support our school's emphasis on technology. 


5.00 


.00 


13. 


Teachers receive adequate administrative support to integrate technology into classroom 
practices. 


5.00 


.00 


15. 


Our school has a well-developed technology plan that guides all technology integration 
efforts. 


4.75 


.50 


17. 


Teachers in this school are generally supportive of technology integration efforts. 


4.50 


.58 




Overall 


4.81 


.24 


Technical Support 


1. 


Most of our school computers are kept in good working condition. 


3.75 


1.26 


2. 


1 can readily obtain answers to technology-related questions. 


4.00 


.00 


6. 


My students have adequate access to up-to-date technology resources. 


4.50 


.58 


7. 


Materials (e.g., software, printer supplies) for classroom use of computers are readily 
available. 


4.50 


.58 




Overall 


4.19 


.38 
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Category and Related TTQ Items 


Means 


Standard 

Deviations 


Attitude toward the Laptop Program 






21 . I feel enthusiastic about the laptop program. 


5.00 


.00 


22. 1 feel my teaching benefited from laptop use this year. 


4.75 


.50 


23. 1 feel confident about my ability with the laptop. 


5.00 


.00 


24. 1 feel like 1 need to learn more skills before 1 can effectively use the laptop for teaching.* 


2.25 


.50 


25. Laptop use has essentially changed my teaching. 


4.00 


.82 


Overall 


4.20 


.28 



* Item reversed. 



Technology Skills Assessment (TSA) 

The primary purpose for the TSA was to assess teacher perceptions of their 
technology ability. The survey begins by asking the teachers to rate "How 
easily..." (Not at all, Somewhat, Very Easily) they could use or complete 47 
computer-related tasks divided into six basic areas: computers, software, 
multimedia, Internet, advanced skills, and using technology for learning. A 
summary of the results from the four fifth grade teacher surveys is presented in 
Table 1 0. 

Teacher confidence was high in all six areas with mean scores of 2.5 or higher, 
between Somewhat and Very Easily. The teachers rated themselves highest in 
Computer Basics (2.98) and Software Basics (2.96). Remarkable is that of the 47 
questions on the TSA, the teachers rated their confidence in 30 of the tasks very 
easily, which constitutes 63.8% of the tasks. Moreover, of the 47 questions, only 4 
questions (# 23, 44, 45 and 46) were rated below 2 ("Somewhat"). 
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Table 10. Technology Skills Assessment Data Summary 



Category and Related TSA Items 


Means 


Standard 

Deviations 


Computer Basics: How easily can you ... 


1 .Use a spell check tool. 


3.00 


.00 


2. Create basic computer documents (word processed) in a timely manner. 


3.00 


.00 


3. Use help menus for software programs. 


2.75 


.50 


4. Use basic computer terms like mouse, keyboard, hard drive, CD-ROM, and monitor. 


3.00 


.00 


5. Save documents so they can be opened on both a Macintosh and PC. 


3.00 


.00 


6. Create folders on a hard drive or disk. 


3.00 


.00 


7. Save files to specific folders. 


3.00 


.00 


8. Locate and delete unwanted files. 


3.00 


.00 


9. Use keyboard commands to cut, copy, or delete text. 


3.00 


.00 


1 0. Proficiently use a mouse and keyboard. 


3.00 


.00 


1 1 .Print a document using "Print" from the File menu and/or the toolbar icon. 


3.00 


.00 


Computer Basics: Overall 


2.98 


.05 


Software Basics: How easily can you ... 


12. Use software preview features to check work. 


3.00 


.00 


1 3. Open and use software programs that are installed on your computer. 


3.00 


.00 


14. Work with and move between two open programs (e.g., Internet and database) to create a product. 


2.75 


.50 


1 5. Describe the difference between downloading and installing software. 


3.00 


.00 


1 6. Save documents so they can be opened in a different program (e.g., from Word to Word Perfect). 


3.00 


.00 


1 7. Install software. 


3.00 


.00 


Software Basics: Overall 


2.96 


.08 


Multimedia Basics: How easily can you ... 


1 8. Import digital video from a camera to a computer. 


3.00 


.00 


1 9. Record and save your voice onto a computer. 


2.67 


.58 


20. Use a scanner to import a photo or document into a computer. 


2.50 


.58 


21 .Play a music CD on the computer. 


3.00 


.00 


Multimedia Basics: Overall 


2.81 


.24 


Internet Basics: How easily can you ... 


22. Connect to the Internet with a modem (phone, cable). 


3.00 


.00 


23. Use Boolean strategies for Internet searches. 


1.75 


.50 


24. Use appropriate software and the Internet to find audio, video, and graphics for lesson plans. 


2.75 


.50 


25. Use the Internet to find help when you have a computer problem. 


2.25 


.96 


26. Determine if information you find on the Internet is accurate and valid. 


3.00 


.00 


27. Evaluate Internet search strategies to determine those that are most efficient. 


3.00 


.00 


28. Determine the usefulness and appropriateness of digital information. 


2.50 


.58 


Internet Basics: Overall 


2.61 


.21 
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Category and Related TSA Items 


Means 


Standard 

Deviations 


Advanced Skills 


29. Use more advanced computer terms like megahertz, gigabytes, and RAM. 


3.00 


.00 


30.Access information on local area networks (LANs) and wide area networks (WANs). 


3.00 


.00 


31 .Use appropriate digital layout and design to meet the needs of defined audiences. 


2.75 


.50 


32. Use appropriate digital layout and design for the selected media (e.g., multimedia, web, print). 


2.75 


.50 


33. Publish information in a variety of media (e.g., printed, monitor display, web-based, video). 


2.75 


.50 


34.Connect a computer to a local server to share files. 


3.00 


.00 


35. Determine if a software program works with an operating system. 


2.50 


.58 


36. Print to a specific printer when connected to a network that has more than one printer. 


3.00 


.00 


37. Use presentation software to share information with specific audiences. 


3.00 


.00 


Advanced Skills: Overall 


2.86 


.11 


Using Technology for Learning 


38. Use multimedia software to enhance learning experiences. 


3.00 


.00 


39. Use appropriate software (e.g., word processing, graphics, databases, spreadsheets, simulations, and 
multimedia) to express ideas and solve problems. 


3.00 


.00 


40. Use text and graphics to create and modify solutions to problems. 


3.00 


.00 


41 .Use digital audio and video to create and modify solutions to problems. 


2.25 


.50 


42. Use communication tools to participate in group projects. 


3.00 


.00 


43. Manipulate information in interactive digital environments (e.g., simulations, virtual labs, field trips). 


2.50 


.58 


44. Participate in a listserv, chat, and bulletin board session. 


1.75 


.96 


45.Create an electronic teaching portfolio to evaluate your work. 


1.75 


.96 


46. Evaluate electronic portfolio products. 


1.75 


.96 


47.Create technology tools to assess student work (e.g., checklists, timelines, rubrics). 


3.00 


.00 


Using Technology for Learning: Overall 


2.50 


.38 
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Focus Groups 

Focus groups were conducted with both teachers and students at the beginning of 
the Learning Without Limits program initiative in fall 2003 and again at the 
conclusion of the academic year in May 2004. Interview questions centered on 
three topic areas: (a) uses of technology, (b) expectations of the initiative and (c) 
reservations with the initiative. From audio recordings and facilitator notes, themes 
were derived from the participants. These are presented below as well as discussed 
later to corroborate and answer the primary evaluation questions. Verbatim 
comments are enclosed in quotation marks to represent most accurately the voice 
of the students and teachers. 

Teacher Reactions 

Computer use. The teachers identified online tests, publishing stories, information 
seeking and research on the Internet, word processing, electronic presentations, 
and draw/paint applications as the computer uses they had implemented. They 
also considered it their responsibility to teach the students about trustworthiness 
with the laptops and the "upkeep" for the laptops. The teachers were proud of the 
very few numbers of computers that had been "dropped" by the students. They 
also felt in order to leverage the use of the laptops, it was necessary to teach 
"saving," "how to save" and management routines for using the laptops. 

The teachers described that they would like to use the laptops for about "50%" of 
the time, about "two to three hours a day." However, they felt some challenges 
prevented them from achieving this goal. Specifically, the scope of the curriculum, 
as well as state and district standards (i.e., "student performance indicators"), was 
difficult to achieve with and without using the laptops. Changes in the district 
curriculum also made it "difficult to schedule" and plan ahead for the next year. 

Pressures and concerns. In the fall semester, the teachers were concerned about the 
keyboarding skills of the students, as well as "maintenance," "upkeep" and 
"technical support" for using the laptops. However, in the spring semester, the 
teachers voiced that they were "confident" in the use of the laptops for instruction 
and that they could "figure out" most of the technical problems or questions that 
arose. They also noted that they depended on one another for expertise. Proximity 
to one another's classrooms facilitated this learning community. 

Similarly, in the fall semester, they described that their colleagues were "jealous" 
toward the fact that they have laptops; however, their colleagues were glad that 
they "did not have to deal with the responsibilities and/or tasks" that accompany 
using the laptops for learning. In the spring, this perception continued. One 
teacher described it as "laptop envy." They felt that some teachers perceived the 
Learning Without Limits program as an "extra burden." Indicative of their growth 
and comfort levels, however, the teachers noted that colleagues "come to us for 
questions" about using computers or technical problems. 
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In the fall semester, the teachers expressed "meeting expectations of the 
administration" as pressure toward using and "meeting standards" as pressures they 
felt toward not using the laptops. However, in the spring semester they seemed to 
have reconciled these. The teachers concurred that they did not feel "as much 
pressure as in the beginning." Some pressure may have been intrinsic — applied by 
the teachers — in addition to the extrinsic pressures. 

In the fall semester, concerns about the laptop program centered on covering 
content and teaching skills and knowledge that specifically may not be revisited 
until later grades. The teachers were also concerned that they would be teaching 
computer skills that the students "will not use in other grade levels." In the spring, 
concerns shifted to improvements for the following school year. For example, 
another teacher would be added to the fifth grade. Currently, the core subject 
areas (language arts, mathematics, science and social studies) were taught by one 
of each of the fifth grade teachers. So, all the fifth grade students rotated through 
each of the teachers. In fall 2004, another teacher will be added and she will most 
likely share teaching responsibilities for one or more content areas. Integrating this 
teacher into the learning community and "logistical ly figuring it out" how to use 
the laptops when the same subject matter is taught in two different classrooms were 
indicated as challenges and concerns. In addition, technical difficulties, such as 
reliability with internal network, external Internet and server access, continue to be 
concerns for the teachers. They do feel, however, that they are more sensitive to 
these challenges given their reliance on these resources. 

Student Reactions 

Computer use. The students identified electronic presentations, writing "stories," 
graphing such as an ordered pairs lesson in mathematics, draw/paint to create an 
original flower in science and visiting Internet sites for information seeking/research 
as ways they had used the laptop computers in class. They said in many instances 
using the laptop computers was "easier," "fun and faster" and required less 
"writing." But they also recognized that in some in cases, such as with the flower 
project in science, it was "harder" than completing the assignment on paper, where 
they previously used craft materials. 

The students reported issues or concerns that they felt prevented from using them 
more in class. These related to technical and logistical topics as well as personal 
responsibility. Technical problems, such as system crashes, "freezing," glitches in 
the application programs, system updates, as well as broken keypads and broken 
cords requiring "repair time," continued to be challenges throughout the school 
year. Logistical issues primarily centered on lack of "battery power," which 
sometimes caused delays in instruction and in a few instances resulted in lost work. 

A very strong theme for the students' use was personal responsibility for the 
laptops. This message as described by the teachers in their interviews appeared to 
have been respected by the students. Many students felt that they "had to be 
careful when using the laptops" so they would not damage them. They also felt 
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that "not being responsible" would prevent them from being able to use them. 
Likewise, "dropping the laptops" was also a concern for the students. But, they 
also felt like they were "gaining responsibility" with using the laptops and taking 
care of them. 

At the beginning of the school year, students thought typing, or keyboarding, skills 
were a problem for students, preventing use or slowing use. They even suggested 
requiring "a typing class." However, at the end of the year, the students did not 
include this with their challenges to using the laptops or with the students they felt 
benefited most from the laptops' use. 

It is also interesting to note that a few students also felt that they "can't work on 
[the laptops] all day." They felt it just wasn't possible. Similarly, students thought 
if they used the laptops "all day" it "might get boring." 

Students that benefit from using laptops. The fifth graders felt that the types of 
students who benefited most from using the laptops were those that were fast 
learners or "students who learn more." They also thought those peers that had 
prior knowledge about computers or were '‘into computers" also benefited. In the 
fall they felt that students who had "high IQs" benefited, but in the spring, students 
mentioned that "students who are not as smart" were the ones benefiting most. 

The students agreed in the fall and spring that the type of students who did not 
benefit from using the laptops were those students who "don't care" and those who 
"don't listen" or do not pay attention. One student called these students "goofers:" 
they "sit there and do nothing." They also felt that the "smartest kids in class" 
benefited least, because "they already know" how to use the laptops and are 
confident with the content. Similarly, the "fast learners" they felt should have 
additional resources, because they are "held back" when skills are retaught. 
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CONCLUSIONS & RECOMMENDATIONS 

The conclusions of the study will be presented in association with each of the 
major research questions in the respective sections below. Recommendations are 
also included, where appropriate, to suggest improvements to the program. 

In what ways has the effectiveness of instruction through the use of student laptop 
computer been impacted ? 

While it is difficult to determine increases without baseline, or beginning, data, 

SOM results indicate extensive uses of cooperative/collaborative learning, project- 
based learning and the teachers acting as coaches or facilitators. Results from the 
SCU indicate extensive uses of productivity tools, specifically draw/paint/graphics 
and electronic presentations, and Internet research with Internet browsers. SCU 
results also suggest wide use across the content areas. Finally, the overall 
meaningfulness of the computers was observed to be extensive in approximately 
one-third to fifty percent of the classroom visits. The results from the SOM coupled 
with these results from the SCU point to activities that result in meaningful uses of 
computers that were based on meaningful problems, required critical thinking skills 
and used computer applications to locate, process and/or manipulate information. 
Despite the limited scope of technology tools, those tools observed were seen to 
meaningfully integrate technology to enhance student learning. 

Moreover, data from the RSCA revealed that teachers used technology with over 
40% of their student-centered learning activities. This finding is not surprising 
given that the observations were conducted with targeted lessons, where the 
teachers were asked to demonstrate technology integration. However, specifically 
notable is that technology was observed in every instance with project-based 
learning. This is a significant accomplishment given the focus of the initiative on 
using technology to impact teaching and learning. Plus, technology for student use 
is often employed best during more unstructured learning such as projects. 

It is important to indicate that during two of the observations as noted in the SCU, 
the school network access was intermittent. As the Learning Without Limits 
program continues and potentially expands, school network reliability will affect 
the effectiveness of instruction with the increased dependence on digital resources, 
such as the Internet, email and school servers. 

To what degree and in what ways have teachers integrated technology with 
classroom instruction ? 

The original Learning Without Limits program included directed technology 
integration training for the fifth grade teachers; this was not implemented due to 
cost and lack of grants support. While no extensive Learning Without Limits 
training was conducted, the fifth grade teachers participated in a workshop on 
using Microsoft PowerPoint in Fall 2003 and informal discussions about classroom 
management with laptops during Spring 2003. Moreover, the teachers through 
focus group interviews indicated they had participated in professional development 
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workshops offered through Shelby County Schools. They also relied on one 
another to extend their expertise, creating a learning community. So, primarily the 
teachers used their educational philosophies and pedagogy to envision effective 
technology integration. 

As discussed above, SOM and SCU data indicate the teachers implemented 
instructional activities that resulted in meaningful uses of computers and required 
critical thinking skills. Productivity tools (e.g., word processing, 
draw/paint/graphics, electronic presentations and Timeliner®) and Internet research 
tools (e.g., Netscape Navigator) were used in all subject areas and educational 
software (e.g., drill and practice) was used in Mathematics. In addition, SOM and 
RSCA data revealed that the teachers employed project-based learning during all 
(1 00%) of the targeted classroom visits to varying degrees (limited to strong 
applications) and technology was used in every instance (100%). These results are 
suggestive that the Learning Without Limits program is effectively impacting 
technology integration efforts at Riverdale Elementary. 

To extend the teachers' effectiveness and expertise with technology integration and 
the instructional strategies that support technology, the teachers should participate 
in professional development targeted at pedagogy and technology integration, such 
as the training proposed in the original Learning Without Limits proposal plan. This 
training may offer the teachers a broader vision for technology integration with the 
school curricula and provide added support for changing teaching and learning 
with laptops. 

To what degree do Learning Without Limits teachers use methodologies that stress 
higher-order learning and student-centered learning activities? 

In almost 90% (88.9%) of the targeted classroom visits, teachers were observed 
extensively to be acting as a coach or facilitator of learning. Other activities 
indicative of critical thinking and student engagement were seen in over 30% of 
the visits. Cooperative/collaborative learning, which was observed extensively in 
33.3% of the visits, was observed to be a somewhat strong or strong application in 
over 40% of the observations (44.4%). Project-based learning, which was 
observed in 1 00% of the visits, was observed to be somewhat strong or strong 
application in almost 75% of the visits (77.7%). Finally, independent 
inquiry/research, which was observed in over 40% of the visits (44.4%), was 
observed to be a somewhat strong or strong application in approximately 1 0% of 
the visits (1 1 .1 %). These data indicate some use of non-traditional, or more 
student-centered, instructional methods. However, professional development 
efforts or goals focus more on these strategies to increase the frequency with which 
they are used. 

To what degree has the Learning Without Limits program impacted teacher 
attitudes toward technology? 

The fifth grade teachers are enthusiastic about the Learning Without Limits program. 
Succinctly, the teachers feel the program has positively impacted their classroom 
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instruction and positively impacted the fifth grade students. Moreover, the teachers 
feel they are ready to integrate technology into their instruction. This is 
corroborated by the focus group interviews as the teachers discussed their 
"confidence" with the laptops and reconciliation with previous intrinsic and 
extrinsic pressures. The TSA highlighted the teachers' expertise in computer basics, 
software basics, multimedia basics, Internet basics, advanced skills and using 
technology to support learning. Personal professional development objectives or 
grade level professional development goals should consider the few technology 
skills, such as Boolean strategy searches, electronic communications other than 
email and electronic teaching portfolios, that received mean scores below 2 
(between "Not at All" and "Somewhat"). These skills should be evaluated for their 
value to the teachers, students and relevance to the program as the initiative 
continues. 

Finally, the teachers felt they have the support of parents, the community, the 
administration and the technical support necessary to be effective with technology 
integration and improve student learning. 

What factors appear most instrumental in determining schools' success at 
achieving the goals and overall implementation of the Learning Without Limits 
program ? 

From the results of the observation instruments (i.e., SOM, SCU, RSCA), teacher 
surveys (i.e., TTQ, TSA) and focus group interviews (with both teachers and 
students), the fifth grade teachers and students are making marked progress to their 
goals. Below is a list of key factors from this evaluation that appear to have 
influenced the progress being made toward achieving the Learning Without Limits 
program implementation and goals. These factors represent high frequencies of 
occurrence and quality of implementation in observations. As seen, these factors 
cross key elements that are critical for program success: 

■ Types of instructional practices 

- Use of student-centered learning 

- Use of technology as a learning tool 

■ Quality of instructional practices 

- Use of meaningful and very meaningful computer activities 

■ Teacher attitudes and beliefs regarding technology integration 

- Teachers feel ready to integrate technology 

- Teachers believe technology positively impact classroom instruction and 
student learning 

- Teachers agreed technology efforts are well-supported 

■ Teacher technology skills 

- Teachers reported confidence with technology tasks 

- Teachers implemented lessons that used a variety of technology tasks 
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Overall, the triangulation of positive results suggest a strong synergy among the 
diverse factors that appear to have contributed to the Learning Without Limits goals 
during the 2003-2004 school year. The fifth grade teachers and administration 
should be proud of the progress the program has made in a short time. As seen, the 
school had positive teacher technology competence and confidence, used 
instructional strategies that centered on and facilitated student learning and 
employed classroom practices that engaged students in meaningful technology- 
supported activities. 
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